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Measuring  Acquisition  Workforce  Quality  Through 
Dynamic  Knowledge  and  Performance:  An  Exploratory 
Investigation  to  Interrelate  Acquisition  Knowledge  With 
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Abstract 

The  efficacy  of  defense  acquisition  is  highly  dependent  upon  acquisition  workforce  quality, 
but  assessing  such  quality  remains  a  major  challenge,  particularly  given  the  knowledge- 
intensive  and  dynamic  nature  of  acquisition  organizations  and  processes.  Hence,  it  is  difficult 
to  gauge — much  less  predict — the  impact  of  leadership  interventions  in  terms  of  policy, 
process,  regulation,  organization,  education,  training,  or  like  approaches.  Building  upon  the 
development  and  application  of  Knowledge  Flow  Theory  (KFT)  over  the  past  couple  of 
decades,  we  have  developed  a  state-of-the-art  approach  that  enables  us  to  analyze, 
visualize,  and  measure  dynamic  knowledge  and  performance.  The  main  idea  is  to  apply  this 
approach  inwardly  to  interrelate  the  knowledge  and  performance  of  acquisition  processes 
(e.g.,  within  contracting  and  project  management  organizations).  In  this  exploratory  study,  we 
examine  acquisition  from  the  perspective  of  the  procurement  process,  focusing  in  particular 
on  organization  knowledge  and  performance  with  respect  to  the  processes  used  for  the 
procurement  of  major  systems  and  services.  We  begin  with  a  summary  of  KFT  and 
measurement  and  then  introduce  the  Contract  Management  Maturity  Model  as  an  approach 
to  acquisition  performance  measurement.  We  follow  in  turn  by  summarizing  the  research 
method  guiding  the  study,  after  which  we  present  preliminary  results  of  our  investigation.  By 
interrelating  knowledge  to  performance  in  terms  of  process  maturity,  this  report  presents  the 
premier  cause-effect  relationship  of  its  kind  in  the  acquisition  domain.  This  article  concludes 
with  key  observations,  limitations,  and  an  agenda  for  continued  research  along  these  lines. 

Introduction 

Acquisition  is  big  business.  The  U.S.  Department  of  Defense  (DoD)  alone  routinely 
executes  12-figure  budgets  for  research,  development,  procurement,  and  support  of  weapon 
systems  and  other  military  products  and  services  (Dillard  &  Nissen,  2005).  Acquisition  is 
also  a  knowledge-intensive  business.  In  addition  to  myriad  laws  governing  federal 
acquisition  in  the  U.S.,  a  plethora  of  rules  and  regulations  specify — often  in  great  detail — 
how  to  accomplish  the  planning,  review,  execution,  and  oversight  of  defense  acquisition 
programs,  large  and  small,  sole-source  and  competitive,  military  and  commercial  (Dillard, 
2003). 
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As  a  result  in  part — and  due  to  high  complexity,  multiple  stakeholders,  goal 
incongruence,  open  process  execution,  and  large  pecuniary  rewards  for  some  participants — 
acquisition  has  been  a  problematic  business,  too.  Seemingly  every  decade,  acquisition 
problems  must  be  addressed  by  another  Blue  Ribbon  panel  and  reformed  yet  again.  The 
Better  Buying  Power  Initiatives  (BBPI),  as  a  recent  instance,  mandated  efficiency  and 
productivity  improvements  in  five  acquisition  business  areas:  (1)  affordability  and  cost 
growth,  (2)  productivity  and  innovation  in  industry,  (3)  competition,  (4)  tradecraft  in  services 
acquisition,  and  (5)  non-productive  processes  and  bureaucracy  (Office  of  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  [OUSD(AT&L)],  2010). 
These  initiatives  focus  principally  on  incentives  for  and  interactions  with  contractors.  The 
Defense  Acquisition  Workforce  Improvement  Act  (DAWIA),  as  another  instance,  was  signed 
into  law  in  1990  and  emphasizes  the  education,  training,  and  certification  of  people  in  the 
acquisition  workforce  (AWF).  Of  course,  the  two  leadership  interventions  are  related:  People 
in  the  AWF  need  to  know  how  to  effect  the  kinds  of  efficiency  and  productivity  improvements 
mandated  via  the  BBPI. 

These  characteristics  of  acquisition  emphasize  the  criticality  of  quality  in  the  AWF 
itself:  With  so  much  at  stake,  and  in  such  a  knowledge-intensive  environment,  a  high-quality 
workforce  is  essential  to  competent  and  professional  acquisition  performance. 

These  characteristics  also  elucidate  the  central  role  played  by  people  and 
organizations  in  the  AWF:  People  must  be  knowledgeable  and  work  effectively — not  only  in 
terms  of  their  own  professional  acquisition  activities  but  also  with  many  others  in  acquisition 
and  customer  organizations — in  order  to  accomplish  key  objectives  and  ensure  timely, 
affordable,  and  responsive  delivery  of  products  and  services  to  fighting  and  support  units,  at 
home  and  abroad.  Indeed,  we  understand  well  how  the  efficacy  of  defense  acquisition  is 
inextricably  dependent  upon  workforce  quality.  Hence,  leadership  interventions  along  these 
lines  appear  to  be  highly  appropriate  and  on  target. 

Assessing  the  impact  of  interventions  such  as  these  is  a  challenge,  however 
(Assistant  Secretary  of  the  Navy  for  Research,  Development,  and  Acquisition  [ASN(RDA)], 
2011a,  2011b).  It’s  unclear  whether  the  relatively  recent  BBPI,  for  instance,  have  had 
sufficient  time  to  produce  measurable  impact.  Even  after  two  decades  of  the  DAWIA,  as 
another  instance,  efficacy  remains  challenging  to  assess,  for  many  extant  measures  (e.g., 
number  of  Defense  Acquisition  University  graduates,  procurement  lead-times,  program  cost 
growth)  fail  to  account  for  critical  aspects  of  the  AWF  and  important  impacts  on  acquisition 
performance.  Indeed,  it  is  difficult  to  gauge — much  less  predict — the  impact  of  any 
leadership  interventions  along  these  lines  (e.g.,  how  much  better  the  AWF  has  become,  or 
even  if  it  is  improving  over  time).  Hence,  the  impact  of  any  particular  leadership  intervention 
is  left  largely  to  anecdote  and  optimism.  To  help  trim  acquisition  budgets  and  guide 
leadership,  an  improvement  in  assessing  leadership  initiatives  and  interventions  is  needed. 

Since  acquisition  is  a  knowledge-intensive  endeavor  (Snider  &  Nissen,  2003),  the 
knowledge  stocks  of  people  comprising  the  AWF  represent  likely  indicators  of  quality  (e.g., 
education  levels,  training  courses,  years  of  experience,  certification  levels).  However,  such 
indicators  are  relatively  static,  pertaining  to  levels  of  knowledge  that  change  comparatively 
slowly  (Nissen,  2006a).  In  contrast,  acquisition  laws,  rules,  and  regulations  are  revised 
frequently,  and  acquisition  knowledge  can  change  abruptly  and  render  obsolete  even  huge 
stocks  overtime.  Indeed,  this  dynamic  acquisition  environment  requires  members  of  the 
AWF  to  sustain  career-long  learning  and  knowledge  development  just  to  remain  proficient 
as  acquisition  professionals.  Thus,  as  indicators  of  AWF  quality,  static  knowledge  stocks 
appear  to  be  out  of  phase  with  the  highly  dynamic  nature  of  the  acquisition  environment. 
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Moreover,  acquisition  organizations  experience  persistent  flux  (Snider  &  Nissen, 
2003).  We  understand  well  that  no  two  acquisition  projects,  programs,  organizations, 
customers,  or  requirements  are  completely  alike.  Hence,  even  well-educated  and  well- 
trained  people,  with  appropriate  certification  levels  and  years  or  decades  of  acquisition 
experience,  must  continually  learn  afresh  and  expand  their  knowledge  further  with  each  new 
assignment.  Likewise,  it  is  clear  that  most  acquisition  organizations  form  and  reform  with 
new  people  (e.g.,  via  personnel  transfer,  turnover,  retirement,  promotion)  continuously  and 
that  end-customer  needs  shift  perennially  (especially  at  the  tactical  edges  of  warfare 
organizations).  Due  to  such  discontinuous  membership  (Ibrahim  &  Nissen,  2007),  even 
these  educated,  trained,  certified,  and  experienced  people  must  learn  repeatedly  to  trust 
and  work  effectively  with  many  others — each  time  someone  new  joins  or  leaves  a  particular 
acquisition  organization,  and  each  time  a  novel  product,  service,  or  customer  is  involved. 
Thus,  dynamic  knowledge  also  appears  to  be  an  important  AWF  quality  indicator. 

The  research  discussed  here  recognizes  these  persistent — and  seemingly 
immutable — attributes  of  the  acquisition  domain  and  workforce,  and  it  seeks  to  overcome 
the  limitations  inherent  in  current  approaches  to  assessing  acquisition  leadership  initiatives 
and  interventions.  Specifically,  this  work  augments  extant,  largely  static  measures  of  AWF 
quality  through  dynamic  knowledge  and  performance  metrics  applied  to  the  acquisition 
domain. 

Building  upon  the  development  and  application  of  Knowledge  Flow  Theory  (KFT) 
over  the  past  couple  of  decades  (Nissen,  2006b),  including  very  recent  work  to  measure 
dynamic  knowledge  and  performance  at  the  tactical  edges  of  military  combat  organizations 
(Nissen  &  Gallup,  2012),  and  applying  such  work  to  the  acquisition  domain  (Nissen,  2012), 
we’re  able  now  to  analyze,  visualize,  and  measure  dynamic  knowledge  flows,  and  we  seek 
to  leverage  such  ability  to  explain  and  predict  corresponding  organization  performance 
levels.  Two  fundamental  research  questions  follow  accordingly: 

•  How  can  dynamic  knowledge  and  performance  metrics  be  applied  to  assess 
AWF  quality? 

•  How  can  knowledge  be  linked  to  measure  and  predict  performance  levels  of 
acquisition  organizations? 

In  this  exploratory  study,  we  examine  acquisition  from  the  perspective  of  the 
procurement  process,  focusing  in  particular  on  organization  knowledge  and  performance 
with  respect  to  the  processes  used  for  the  procurement  of  major  systems  and  services, 
ranging  from  research  and  development  to  weapon  system  equipment  and  related 
maintenance.  We  begin  with  a  summary  of  KFT  and  measurement  and  then  introduce  the 
Contract  Management  Maturity  Model  (CMMM;  see  Rendon,  2003)  as  an  approach  to 
acquisition  performance  measurement.  We  follow  in  turn  by  summarizing  the  research 
method  guiding  the  study,  after  which  we  present  preliminary  results  of  our  investigation.  By 
interrelating  knowledge  to  performance  in  terms  of  process  maturity,  this  report  presents  the 
premier  cause-effect  relationship  of  its  kind  in  the  acquisition  domain.  This  article  concludes 
with  key  observations,  limitations,  and  an  agenda  for  continued  research  along  these  lines. 

Knowledge  Flow  Theory  and  Contract  Management  Maturity 

In  this  section,  we  begin  with  a  summary  of  KFT  and  measurement  and  then 
introduce  the  CMMM,  discussing  the  potential  for  linking  and  using  these  two  frameworks  to 
assess  acquisition  performance. 
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Knowledge  Flow  Theory  and  Measurement 

The  dynamic  nature  of  knowledge  indicates  that  both  stocks  and  flows  are  important 
(Dierickx  &  Cool,  1989).  Knowledge  stocks  have  been  comparatively  straightforward  to 
measure  historically;  metrics  pertaining  to  education  levels,  training  courses,  years  of 
experience,  certifications,  and  like  knowledge-oriented  factors  are  employed  broadly. 
Alternatively,  knowledge  flows  have  been  comparatively  much  more  difficult  to  assess; 
metrics  pertaining  to  dynamic  knowledge — particularly  at  the  group  and  organization 
levels — are  more  elusive.  The  development  and  application  of  KFT  (e.g.,  see  Nissen, 
2006b)  over  the  past  couple  of  decades  has  augmented  the  set  of  tools  and  techniques 
available  to  analyze,  visualize,  and  measure  dynamic  knowledge  and  performance  in  the 
organization. 

KFT  is  founded  on  a  set  of  30  principles  that  characterize  dynamic  knowledge.  Such 
principles  are  actionable  and  empirical,  and  they  support  the  diagnosis  of  workflow  and 
knowledge-flow  process  pathologies,  visualization  of  improvement  interventions,  and 
measurement  of  dynamic  knowledge  and  performance  gains  (Nissen,  2006a).  Dynamic 
knowledge  is  delineated  via  five-dimensional  (5D)  vector  space.  Knowledge-flow  vectors 
carry  measurements  and  elucidate  diagnostic  inferences  pertaining  to  the  people, 
processes,  and  organizations  associated  with  knowledge  work.  Figure  1  illustrates  the  idea. 


Figure  1.  5D  Knowledge-Flow  Diagram 

Briefly,  the  vertical  axis  “Explicitness”  characterizes  the  nature  of  knowledge  along  a 
tacit-explicit  continuum.  Tacit  knowledge  implies  understanding  and  know-how/why,  and  it 
is  associated  most  closely  with  the  experiences  of  people  (e.g.,  stemming  from  job 
assignments,  mentoring,  and  teamwork)  and  routines  of  organizations  (e.g.,  culture, 
process,  ritual).  Explicit  knowledge  implies  awareness  and  know-who/what/where/when,  and 
it  is  associated  most  closely  with  artifacts  (e.g.,  documents,  formulae,  software).  Generally, 
the  more  tacit  the  knowledge,  the  greater  its  appropriability  and  potential  impact  on  positive 
performance  becomes  (Saviotti,  1998).  One  can  measure  knowledge  explicitness  using 
ordinal,  interval,  or  ratio  scales. 
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The  horizontal  axis  “Reach”  characterizes  how  broadly  knowledge  is  known  and 
shared  in  an  organization.  Here  we  operationalize  reach  in  terms  of  the  number  of  people  in 
an  organization  who  have  access  to  and  can  employ  any  particular  chunk  of  knowledge,  but 
we  could  view  reach  in  terms  of  organization  levels  instead  (e.g.,  individual,  group, 
organization,  interorganization).  Generally,  the  broader  the  reach  of  knowledge,  the  greater 
its  amplification  and  potential  impact  on  positive  performance  becomes  (Nonaka,  1994). 
Measurements  can  be  made  using  ordinal,  interval,  or  ratio  scales. 

The  axis  “Life  cycle”  characterizes  what  is  being  done  with  a  particular  chunk  of 
knowledge  at  some  specific  point  in  time.  Here  we  include  three  activities:  (1 )  some 
individual  in  the  organization  learns  or  creates  new  knowledge;  (2)  he  or  she  shares  existing 
knowledge  with  or  transfers  it  to  other  people  in  the  organization;  and  (3)  one  or  more 
people  in  the  organization  use  or  apply  existing  knowledge  to  accomplish  work.  Generally, 
knowledge  does  not  become  useful  until  it  is  used  or  applied  (Pfeffer  &  Sutton,  1999). 
Measurements  can  be  made  using  categorical  or  ordinal  scales. 

Because  visualization  beyond  three  dimensions  is  difficult,  we  represent  the 
dimension  “Flow  time”  in  terms  of  the  thickness  of  lines  used  to  delineate  vectors.  As  shown 
in  the  key  to  the  right  of  Figure  1 ,  relatively  thin  lines  are  used  to  delineate  short  and  fast 
knowledge  flows,  whereas  comparatively  thick  lines  represent  knowledge  that  takes  a  long 
time  and  flows  slowly.  Generally,  the  more  quickly  that  knowledge  flows  (e.g.,  across 
people,  organizations,  places,  times),  the  greater  its  potential  impact  on  positive 
performance  becomes  (Nissen,  2002).  Measurements  can  be  made  using  ordinal,  interval, 
or  ratio  scales. 

The  dimension  “Power”  is  represented  similarly  in  terms  of  line  style  used  to 
delineate  knowledge-flow  vectors.  Knowledge  that  flows  with  relatively  low  power — this 
corresponds  with  relatively  low  performance  levels  of  organization  activities  enabled  by  the 
knowledge — is  delineated  through  orange  dotted  lines,  whereas  knowledge  flows  exhibiting 
high  power — and  hence  enabling  high  performance — are  delineated  via  purple  solid  lines. 
Measurements  can  be  made  using  ordinal,  interval,  or  ratio  scales. 

Integrating  these  five  dimensions  graphically  and  analytically  generates  a  5D  vector 
space  to  examine  dynamic  knowledge.  Such  5D  space  and  examination  schemes  are 
completely  general:  They  can  be  applied  to  any  dynamic  knowledge  in  any  organization 
domain  (e.g.,  acquisition,  command  and  control,  software  engineering). 

As  an  example  of  use  and  application,  consider  Figure  2,  which  illustrates  an 
important  knowledge  flow  desired  by  the  organization.  Point  A  represents  one  individual  in 
the  organization  who  learns  something  new  (to  that  organization)  or  creates  entirely  new 
knowledge.  In  terms  of  the  5D  space,  this  represents  tacit  knowledge  that  is  created  by  an 
individual  (i.e.,  one  person),  hence  its  position  at  the  bottom-back  corner  of  the  diagram. 
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Figure  2.  Knowledge  Creation  and  Application  Needs 

In  the  acquisition  domain,  for  instance,  consider  that  such  new  knowledge  could 
pertain  to  a  technique  for  reducing  the  acquisition  time  for  an  important  information  system 
(IS)  needed  in  the  field.  Because  information  technology  (IT)  advances  so  quickly — 
outpacing  the  ability  of  many  acquisition  organizations  to  develop  and  field  systems 
responsively — the  organization  views  this  new  knowledge  created  at  Point  A  as  important, 
and  it  would  like  to  see  such  knowledge  shared  with  and  applied  by  all  100  people  in  that 
organization  who  work  with  IT. 

Such  application  by  100  people  in  the  organization  is  represented  by  Point  B.  The 
thin,  purple,  solid  vector  connecting  Points  A  and  B  represents  the  desired  knowledge  flow: 
The  organization  wishes  for  such  knowledge  to  flow  quickly  and  with  high  power  (e.g., 
enabling  all  100  people  at  Point  B  to  work,  within  one  day,  at  the  same  performance  level  as 
the  innovative  individual  at  Point  A).  This  represents  a  5D  knowledge-flow  vector.  A  question 
mark  in  the  figure  next  to  the  vector  indicates  that  such  a  fast,  powerful  knowledge  flow  is 
desired  by  the  organization,  but  it  is  unclear  which,  if  any,  organization  process  can  enable 
it.  Indeed,  most  organizations  do  lack  such  a  process  (Nissen,  2006b).  Some  other 
approach  to  sharing  and  applying  the  important  IT  acquisition  knowledge  is  required. 

Figure  3  delineates  two  alternate  archetypical  knowledge  flows  corresponding  to 
processes  that  are  within  this  organization’s  capabilities.  (We  say  archetypical,  because 
most  organizations  employ  these  classic  processes  routinely,  and  because  they  present  a 
vivid  contrast  in  terms  of  how  dynamic  knowledge  flows.)  One  knowledge  flow  is  depicted  in 
terms  of  a  relatively  fast  (i.e.,  thin  lines)  but  low-power  (i.e.,  orange,  dotted  lines)  vector 
series;  this  first  flow  is  associated  with  explicit  knowledge  and  utilizes  one  or  more  ISs  for 
knowledge  articulation  and  distribution  in  explicit  form.  The  other  is  delineated  via  a 
comparatively  slow  (i.e.,  thick  lines)  but  high-power  (i.e.,  purple,  solid  lines)  vector;  this 
second  flow  is  associated  with  tacit  knowledge  and  utilizes  one  or  more  human-centered 
approaches  to  knowledge  sharing  (e.g.,  group  interaction,  mentoring,  personnel  transfer). 
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Figure  3.  Alternate  Archetypical  Knowledge  Flows 

The  key  is  that  one  can  measure  these  five  dimensions  of  knowledge — whether  via 
explicit  or  tacit  flows — and  relate  them  to  the  corresponding  knowledge-based  process 
performance  by  people  in  the  organization.  Indeed,  by  correlating  such  dynamic  knowledge 
measures  with  performance  metrics,  one  can  develop  a  model  capable  of  analyzing, 
visualizing,  and  even  predicting  process  performance  based  upon  knowledge-flow  patterns. 

Of  course,  many  diverse  combinations  of  these  archetypical  knowledge  flows  are 
possible,  too,  yet  most  knowledge  flows  are  likely  to  reflect  some  aspects  of  these  two 
dynamic  patterns  (Nissen,  2006b).  Through  empirical  analysis  and  calibration  of  specific 
knowledge  flowing  through  any  particular  organization  in  the  field,  one  can  correlate  5D 
dynamic  knowledge  flows  with  work  performance,  resulting  in  a  model  capable  of 
measurement  and  prediction.  Through  this  technique,  we  are  working  to  assess  AWF  quality 
in  terms  of  dynamic  knowledge  flows. 

Contract  Management  Maturity  Model 

Contract  management  is  a  notably  challenging  process,  which  can  be  viewed 
productively  via  six  phases:  (1)  procurement  planning,  (2)  solicitation  planning,  (3) 
solicitation,  (4)  source  selection,  (5)  contract  administration,  and  (6)  contract 
closeout/termination  (Rendon,  2008).  Both  individually  and  together  as  a  set,  these  six 
phases  of  the  contract  management  process  form  the  basis  for  assessing  contract 
management  process  capability  and  maturity,  which  offer  in  turn  an  approach  to  measuring 
performance  (Garrett  &  Rendon,  2005;  Rendon,  2008). 

In  a  general  process  sense  not  specific  to  contract  management,  process  capability 
is  defined  as  “the  inherent  ability  of  a  process  to  produce  planned  results”  (Ahern,  Clouse,  & 
Turner,  2001,  p.  4).  As  the  capability  of  a  process  increases,  it  becomes  predictable  and 
measurable.  As  the  organization  steadily  improves  its  process  capability,  organization 
competence  increases,  and  organization  processes  become  more  mature  (Ahern  et  al., 
2001).  Competence,  in  this  case,  is  defined  as  “an  underlying  characteristic  that  is  causally 
related  to  effective  or  superior  performance,  as  determined  by  measurable,  objective  criteria, 
in  a  job  or  in  a  situation”  (Curtis,  Hefley,  &  Miller,  2001,  p.  577).  Maturity  can  be  defined  as 
“a  measure  of  effectiveness  in  any  specific  process”  (Dinsmore,  1998,  p.  169).  It  is  important 
to  note  that  process  maturity  is  not  related  to  the  passage  of  time.  Different  organizations 
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mature  at  different  rates,  depending  on  the  nature  of  the  business  and  the  emphasis  placed 
on  process  improvement.  Process  maturity  is  more  reflective  of  how  far  an  organization  has 
progressed  toward  continuously  improving  its  process  capability  in  any  specific  area. 

Organization  process  capability  can  be  assessed  using  a  process  maturity  model. 
These  maturity  models  are  built  on  a  series  of  maturity  levels — each  maturity  level  reflective 
of  the  level  of  competence  for  that  process.  As  the  organization  gains  process  competence, 
it  moves  up  the  maturity  scale.  As  maturity  increases,  so  does  capability  and  predictability, 
while  risk  decreases.  Rendon  (2003)  was  the  first  to  apply  the  concept  of  process  capability 
and  maturity  to  organization  contract  management  processes.  The  CMMM  was  developed 
as  a  method  for  assessing  an  organization’s  contract  management  process  capability  and 
using  the  assessment  results  to  identify  contract  management  process  deficiencies  and  the 
need  for  process  improvement.  The  CMMM  has  been  applied  at  Air  Force,  Army,  Navy,  and 
defense  contractor  organizations  (Rendon,  2008,  2009,  2010,  2011).  The  structure  of  the 
CMMM  is  based  on  the  six  contract  management  process  phases  previously  discussed  and 
on  the  five  levels  of  contract  management  process  capability  maturity.  Specifically,  the  five 
levels  of  contract  management  process  maturity  consist  of  Level  1  (Ad  Hoc),  Level  2  (Basic), 
Level  3  (Structured),  Level  4  (Integrated),  and  Level  5  (Optimized). 

From  this  discussion,  it  should  become  clearer  how  knowledge  interrelates 
dynamically  with  process  capability  and  maturity.  Not  only  does  knowledge  exist,  grow,  and 
move  within  and  between  individual  participants  of  the  AWF,  manifested  through  their 
professional  performance  of  acquisition  tasks,  but  we  find  it  also  at  the  organization  level, 
manifested  through  the  professional  performance  of  processes,  which  are  accomplished  by 
many  individual  people,  working  together  to  accomplish  many  tasks  toward  a  set  of  shared 
goals.  Not  only  must  individual  people  understand  and  be  able  to  accomplish  the  key  tasks 
comprising  process  work,  but  the  organization  as  a  whole  must  also  understand  and  be  able 
to  accomplish  the  process  itself.  Organizations  at  higher  maturity  levels  have  arguably 
developed  greater  process-level  understanding  than  their  lower  level  counterparts;  hence, 
they  show  potential  to  manifest  correspondingly  higher  process  performance  levels. 
Knowledge — at  the  organization  level  as  well  as  the  individual  level — drives  such 
understanding  and  potential. 

Research  Method 

Building  upon  recent  case  study  research  (Barnes  &  Williams,  2012;  Nissen,  2012), 
in  addition  to  our  general  acquisition  experience  and  CMMM  assessments  (Rendon,  201 1 ), 
we  integrated  data  from  previous  studies  to  establish  a  basis  for  AWF  quality  assessment 
via  KFT-CMMM  linkage.  In  particular,  we  employed  KFT  and  5D  modeling  to  identify  key 
independent  variables  (e.g.,  kinds  and  levels  of  knowledge)  and  CMMM  assessment  results 
to  identify  corresponding  dependent  variables  exhibiting  good  potential  in  our  research 
context.  We  then  explored  the  integration  of  these  two  parts  (i.e.,  matching  the  independent 
variables  with  dependent  variables)  through  a  combination  of  qualitative  and  quantitative 
analysis. 

Results 

Results  from  this  exploratory  investigation  center  on  delineating  the  procurement 
process  via  KFT,  summarizing  corresponding  CMMM  assessments,  and  linking  the  two  to 
elucidate  insight  into  AWF  quality.  We  begin  with  an  overview  of  the  focal  organizations. 

Focal  Organizations 

The  two  focal  organizations  of  this  study  include  the  contracting  centers  at  two  large, 
operational  DoD  organizations.  Using  pseudonyms,  the  Organization  T  contracting  center 
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provides  acquisition  and  contracting  support  for  the  procurement  of  military  equipment  and 
hardware.  This  includes  procuring  research  and  development,  systems,  and  repair  parts  and 
services.  The  Organization  T  contracting  center  does  a  relatively  large  amount  of  business 
(over  $10  billion  annually). 

The  Organization  R  contracting  center  provides  innovative  acquisition  and 
contracting  support  for  research  and  development  acquisition.  The  Organization  R 
contracting  center  also  does  a  relatively  large  amount  of  business  (almost  $7  billion 
annually). 

Although  the  Organization  T  and  Organization  R  contracting  centers  acquire  and 
procure  different  types  of  supplies  and  services,  the  contract  management  processes  used 
are  common  to  both  organizations  (Rendon  &  Snider,  2008).  Additionally,  the  contract 
management  processes  used  at  these  contracting  centers  are  also  common  to  other  DoD 
and  federal  government  agencies  for  the  procurement  of  supplies  and  services.  Thus, 
conclusions  based  on  the  analysis  of  the  results  from  these  contract  management  process 
assessments  may  be  applicable  to  other  DoD  and  federal  government  agencies.  Indeed, 
given  that  many  of  the  same  process  activities  are  required  for  all  procurements,  some 
aspects  of  our  results  should  apply  quite  broadly  and  well  beyond  the  government  sector. 

Analysis 

In  this  section,  we  summarize  some  of  the  key  knowledge  data  associated  with  our 
focal  contracting  organizations.  In  particular,  we  examine  and  compare  three 
complementary  knowledge  proxies:  procuring  contracting  officer  (PCO)  density,  DAWIA 
level,  and  experience.  The  first  measure  characterizes  the  fraction  of  respondents  surveyed 
in  each  organization  that  possess  contracting  warrants.  The  measure  represents  a  sign  of 
knowledge  and  experience  in  addition  to  education.  It  also,  to  some  extent,  represents 
organization  contracting  capacity.  Since  only  warranted  PCOs  can  sign  contract  documents, 
the  greater  the  number  of  PCOs,  the  greater  the  organizations’  capacity  for  performing  the 
contracting  mission.  The  measure  varies  from  0  (i.e.,  no  one  has  a  warrant)  to  1  (i.e., 
everyone  is  warranted).  The  second  measure  characterizes  respondents’  certification  levels. 
As  noted  in  the  previous  section,  there  is  an  educational  component  (e.g.,  bachelor’s  degree 
and  additional  training)  along  with  an  experiential  one  (e.g.,  three  years’  experience).  The 
third  measure  sums  respondents’  years  of  experience  in  the  contracting  domain.  This 
measure  is  comparatively  pure  in  terms  of  reflecting  principally  tacit  knowledge. 

There  is  likely  to  be  some  confounding  and  collinearity  across  these  measures,  as 
they  all  reflect  some  degree  of  experience  (i.e.,  tacit  knowledge),  and  the  first  two  both 
reflect  some  degree  of  education  (i.e.,  explicit  knowledge)  as  well.  Nonetheless,  they  do 
represent  knowledge  proxies  that  lend  themselves  to  examination  in  our  exploratory 
research  context.  Further,  although  each  of  these  three  measures  is  static  in  nature,  all 
three  are  expected  to  have  strong  association  with  organization  performance  in  terms  of  our 
process  maturity  measure.  Follow-on  research  can  build  upon  our  results  to  refine  these 
data,  expand  data  collection,  and  investigate  dynamic  knowledge  measures  as  well. 

Table  1  summarizes  numerical  results  for  the  Organization  T  suborganizations 
individually  and  as  a  whole.  For  Organization  T,  the  CMMM  survey  resulted  in  132 
responses,  reflecting  a  response  rate  of  56%.  The  first  column  identifies  each  individual 
suborganization  by  a  two-letter  code  (e.g.,  “AB”)  and  includes  average  values  across  all 
seven  such  individual  suborganizations  (i.e.,  “AN”).  The  second  column  summarizes  the 
relative  density  of  warranted  contracting  officers  surveyed  in  each  suborganization.  In  the 
AB  suborganization,  for  instance,  roughly  40%  of  survey  respondents  were  warranted  as 
such.  This  exceeds  the  density  of  warranted  contracting  officers  in  the  organization  as  a 
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whole  of  course,  but  it  reflects  the  relatively  high-level  pool  of  people  surveyed.  Such  high- 
level  people  possess  relatively  good  understanding  of  the  organization’s  contracting 
processes  and  hence  serve  as  appropriate  respondents  to  evaluate  its  maturity.  In  similar 
fashion,  the  third  column  summarizes  the  relative  certification  level  in  each  organization.  In 
this  same  AB  suborganization,  for  instance,  the  average  certification  level  is  2.6.  This 
reflects  likewise  appropriately  a  relatively  highly  certified  pool  of  respondents  with  Level  II 
and  Level  III  DAWIA  certifications.  The  fourth  column  summarizes  the  relative  experience 
level  in  each  organization,  with  suborganization  AB  shown  at  nearly  1 1  years’  average 
experience  among  contracting  personnel  surveyed.  Notice  how  the  HD  suborganization  at 
the  bottom  of  the  table  appears  to  be  something  of  an  outlier  with  a  considerably  higher 
experience  level  (i.e.,  16.8  years). 

Table  1 .  Organization  T  Knowledge  Summary 


Org 

PCO 

DAWIA 

Years 

AB 

0.4 

2.6 

10.8 

AD 

0.2 

2.6 

11.4 

AH 

0.2 

2.5 

10.7 

Al 

0.6 

2.8 

12.8 

AS 

0.2 

2.6 

10.8 

AT 

0.3 

2.4 

11.6 

HD 

0.2 

2.7 

16.8 

All 

0.3 

2.6 

12.1 

Note,  n  =  132 


Table  2  summarizes  knowledge  data  in  the  same  manner  for  Organization  R.  Here 
the  RT  suborganization  reflects  that  every  respondent  possesses  Level  III  DAWIA 
certification.  Notice  how  the  overall  comparison  (i.e.,  “All”  data)  across  these  two 
organizations  reflects  slightly  higher  values  for  each  knowledge  proxy  (i.e.,  PCO:  0.5  versus 
0.3;  DAWIA:  2.7  versus  2.6;  years:  13.9  versus  12.1),  whereas  the  individual  organizations 
show  comparative  plusses  and  minuses  across  each  proxy.  For  Organization  R,  the  CMMM 
survey  resulted  in  96  responses,  reflecting  a  response  rate  of  44%. 

Table  2.  Organization  R  Knowledge  Summary 


Org 

PCO 

DAWIA 

Years 

AD 

0.5 

2.5 

14.9 

Al 

0.5 

2.5 

12.3 

AP 

0.5 

2.8 

13.1 

ED 

0.6 

2.7 

15.0 

RT 

0.4 

3.0 

14.1 

All 

0.5 

2.7 

13.9 

Note,  n  =  96 


Continuing  the  discussion,  look  now  at  the  expanded  Table  3,  which  incorporates 
two  additional  columns  of  information  pertaining  to  Organization  T.  The  fifth  column 
summarizes  the  maturity  score  for  each  organization.  This  score  represents  the  sum  across 
all  six  categories,  averaged  for  each  individual  suborganization.  The  AB  suborganization,  for 
instance,  has  a  score  of  221 .6.  This  is  the  sum  of  six  category  scores  for  that 
suborganization  (i.e.,  38.8  for  the  procurement  planning  part,  40.8  for  the  solicitation 
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planning  part,  39.0  for  the  solicitation  part,  36.4  for  the  source  selection  part,  39.8  for  the 
contract  administration  part,  26.8  for  the  contract  closeout  part).  The  score  for  each  category 
represents  the  average  across  all  respondents  in  that  organization.  As  noted  in  the  previous 
section,  higher  scores  correspond  with  higher  maturity  levels.  The  same  scheme  follows  for 
the  other  suborganizations  reported  in  this  table.  Table  4  follows  the  same  format  for 
Organization  R. 


Table  3.  Organization  T  Maturity  Summary 


Org 

PCO 

DAWIA 

AB 

0.4 

2.6 

AD 

0.2 

2.6 

AH 

0.2 

2.5 

Al 

0.6 

2.8 

AS 

0.2 

2.6 

AT 

0.3 

2.4 

HD 

0.2 

2.7 

All 

0.3 

2.6 

Years 

Score 

Maturity 

10.8 

221.6 

2 

11.4 

218.5 

2 

10.7 

202.5 

2 

12.8 

267.5 

4 

10.8 

240.6 

3 

11.6 

221.4 

2 

16.8 

241.7 

3 

12.1 

230.6 

2 

It  is  important  to  note  that  we  generally  calculated,  reported,  and  interpreted  maturity 
scores  for  each  of  the  six  categories  individually  (Rendon,  201 1 ).  Continuing  to  use  the  AB 
suborganization  as  an  example,  we  would  normally  use  its  score  of  38.8  for  the  procurement 
planning  part,  for  instance,  for  conversion  to  a  maturity  level  of  3  (Structured).  We  would 
then  likewise  use  its  scores  of  40.8  for  the  solicitation  planning  part  and  39.0  for  the 
solicitation  part  for  similar  conversion  to  Level  3.  The  scores  of  36.4  for  the  source  selection 
part,  39.8  for  the  contract  administration  part,  and  26.8  for  the  contract  closeout  part  convert 
to  Level  2  (Basic).  Summarizing  up  from  this  level  of  detail,  the  combined  score  of  221.6 
would  convert  to  an  overall  maturity  level  of  2,  only  just  shy  of  Level  3.  This  level  is  reported 
in  the  sixth  column  for  both  the  Organization  T  and  Organization  R  suborganizations.  The 
overall  maturity  levels  of  2  for  both  organizations  as  wholes  reflect  weighted  averages  of 
these  individual  organization  levels. 


Table  4.  Organization  R  Maturity  Summary 


Org 

PCO 

DAWIA 

Years 

Score 

Maturity 

AD 

0.5 

2.5 

14.9 

236.2 

3 

Al 

0.5 

2.5 

12.3 

189.6 

2 

AP 

0.5 

2.8 

13.1 

206.5 

2 

ED 

0.6 

2.7 

15.0 

222.2 

2 

RT 

0.4 

3.0 

14.1 

167.8 

2 

All 

0.5 

2.7 

13.9 

204.5 

2 

Now  we  seek  to  interconnect  and  interrelate  the  two  kinds  of  data  by  exploring  any 
relationships  that  may  be  apparent  between  our  knowledge  proxies  and  maturity  scores.  We 
can  view  these  same  data  graphically.  Figure  4  delineates  the  overall  relationship  between 
PCO  and  maturity  score,  for  instance,  across  all  organizations  combined  (i.e.,  Organization 
T  and  Organization  R  together).  Overall,  this  does  not  appear  to  reflect  a  strong  relationship, 
however.  Indeed,  the  regression  R2  for  this  pair  is  0  (p  =  0.87).  When  combining  the  two 
organizations  together,  no  pairwise  relationship  appears  strong  (e.g.,  R2  =  0.04,  p  =  0.56  for 
Score-DAWIA;  R2  =  0.01 ,  p  =  0.74  for  Score-Years).  We  do  not  plot  these  other  views. 
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Figure  4.  Combined  Score-PCO  Relationship 

The  relationship  becomes  somewhat  more  apparent  when  we  separate  the  two 
organizations,  however.  Looking  solely  at  Organization  T  in  Figure  5  or  Organization  R  in 
Figure  6,  for  instance,  the  interrelationship  is  more  noticeable,  both  graphically  and 
statistically  (e.g.,  Organization  T:  R2  =  0.36,  p  =  0.15;  Organization  R:  R2=  0.41,  p  =  0.25). 


PCO 


Figure  5.  Organization  T  Score-PCO  Relationship 
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Figure  6.  Organization  R  Score-PCO  Relationship 

Figures  7  and  Figure  8  each  delineates  the  respective  relationship  between  DAWIA 
and  maturity  score,  for  instance,  across  the  two  organizations  individually  (i.e.,  Organization 
T  and  Organization  R  separately).  Overall,  this  appears  to  reflect  a  stronger  relationship 
(e.g.,  Organization  T:  R2  =  0.64,  p  =  0.03;  Organization  R:  R2  =  0.44,  p  =  0.22)  than  those 
depicted  previously  in  Figures  5  and  6  for  PCO. 


300  0 

250  0 

5  200  0 
C 

o  isao 


r 

e 


100  0 
50.0 
00 


2.0 


2.2 


OrgT 


♦ 


2.4  2.6 

DAWIA 


2.8 


3.0 


Figure  7.  Organization  T  Score-DAWIA  Relationship 
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Figure  8.  Organization  R  Score-DAWIA  Relationship 

Figure  9  delineates  the  relationship  between  years  and  maturity  score,  for  instance, 
across  Organization  T  individually.  In  this  view,  the  one  outlier  (i.e.,  the  HD  suborganization) 
noted  in  the  previous  section  appears  quite  prominently,  and  it  affects  the  statistical  fit  (e.g., 
R2=  0.22,  p  =  0.29)  accordingly. 


Years 


Figure  9.  Organization  T  Score-Years  Relationship 

When  we  remove  that  outlier,  however,  as  reflected  in  Figure  10,  the  relationship 
between  years  and  maturity  score  becomes  considerably  more  apparent  graphically  and 
stronger  statistically  (e.g.,  R2  =  0.59,  p  =  0.07).  The  corresponding  relationship  is  not  as 
strong  when  viewing  Organization  R  separately  as  in  Figure  1 1  (e.g.,  R2  =  0.27,  p  =  0.37) 
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Figure  10.  Organization  T  Score-Years  Relationship  (Sans  Outlier) 
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Figure  11.  Organization  R  Score-Years  Relationship 

Our  final  look  at  these  relationships  involves  multiple  regression  that  includes  all 
three  knowledge  proxies.  We  understand,  of  course,  that  results  can  be  potentially  highly 
spurious — and  accordingly,  the  results  can  be  potentially  very  tenuous — with  so  few 
observations,  but  it  accommodates  our  exploratory  interest,  and  it  provides  some  new 
insight,  nonetheless.  Given  the  associated  multidimensionality,  we  do  not  show  graphics  for 
this,  but  we  note  the  statistical  results  for  Organization  T  (e.g.,  R2  =  0.71 ,  p  =  0.24)  and 
Organization  R  (e.g.,  R2  =  0.72,  p  =  0.64)  individually.  Noting  the  small  number  of 
observations  again,  these  statistics  suggest  that  knowledge  as  measured  via  our  static 
proxies  corresponds  with  performance  in  terms  of  process  maturity.  This  represents  a 
noteworthy  discovery,  and  it  begins  to  provide  some  empirical  evidence  to  support  our 
theoretical  connection  between  knowledge  flows  and  process  maturity.  We  address  several 
avenues  for  future  research  to  build  upon  these  exciting  and  new  but  exploratory  and 
tenuous  results. 

Conclusion 

In  this  exploratory  study,  we  examined  acquisition  from  the  perspective  of  the 
procurement  process,  focusing  in  particular  on  organization  knowledge  and  performance 
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with  respect  to  the  processes  used  for  the  procurement  of  major  systems  and  services. 
Building  upon  considerable  field  research  involving  two  large,  operational  defense 
organizations,  we  collected  data  to  summarize  the  absolute  and  relative  knowledge  levels 
across  multiple  organization  units  within  both  focal  organizations.  Specifically,  we  identified 
three  knowledge  proxies  with  good  theoretical  potential  to  correspond  well  with 
performance:  PCO  density,  DAWIA  level,  and  experience.  We  also  collected  data  to 
summarize  these  organizations  and  units’  corresponding  process  maturity  levels  as  a 
performance  measure,  and  we  worked  to  interrelate  knowledge  and  performance 
accordingly. 

Examination  of  the  organizations’  data  revealed  considerable  variation,  both  within 
and  across  them,  in  terms  of  all  three  knowledge  proxies,  and  we  also  found  ample  variation 
in  terms  of  process  maturity.  Because  our  unit  of  analysis  is  the  organization  unit,  we  do  not 
have  a  large  number  of  observations  to  support  sophisticated  statistical  analysis.  We  do, 
nonetheless,  provide  both  graphical  and  statistical  summaries  of  the  interrelationships 
between  knowledge  and  performance,  summaries  that  suggest  a  noteworthy  empirical 
correspondence  as  predicted  theoretically.  Further,  by  interrelating  knowledge  to 
performance  in  terms  of  process  maturity,  this  report  presents  the  premier  cause-effect 
relationship  of  its  kind  in  the  acquisition  domain. 

This  article  provides  only  an  exploratory  beginning  to  such  promising  research, 
however;  hence,  it  illuminates  a  number  of  opportunities  for  productive  follow-on  work  to 
build  upon  our  results.  For  one,  promising  follow-on  research  could  leverage  the  method 
developed  here  to  expand  the  study  across  many  other  organizations  and  units.  For 
another,  future  research  to  increase  the  number  and  sophistication  of  such  proxies  could 
lead  to  sharper,  more  discriminatory  measures.  Fora  third,  considerable  follow-on  research 
to  validate  the  survey  instruments,  calibrate  the  conversion  tables,  and  understand  how 
maturity  interrelates  with — and  conceivably  influences — other  performance  dimensions 
appears  to  be  open-ended  and  highly  promising  at  present.  Finally,  a  desirable  end  result 
includes  offering  policy,  leadership,  and  management  guidance  for  the  acquisition 
community.  Given  the  importance  of  knowledge  in  terms  of  workforce  quality,  and  given  the 
dynamic  nature  of  both  the  acquisition  domain  and  its  workforce,  managers,  leaders,  and 
policy-makers  need  better  guidance  to  develop  organization  innovations  that  offer  good 
potential  for  positive  impact.  We  continue  making  good  metaphorical  strides  toward  this 
result,  which  continued  funding  and  research  will  facilitate  and  drive. 
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